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Abstract. The Restricted Isometry Constants (RIC) of a matrix A measures how close to an isometry is the 
action of A on vectors with few nonzero entries, measured in the (? norm. Specifically, the upper and lower RIC of 
a matrix A of size n X N is the maximum and the minimum deviation from unity (one) of the largest and smallest, 
respectively, square of singular values of all ( . ) matrices formed by taking k columns from A. Calculation of the 
RIC is intractable for most matrices due to its combinatorial nature; however, many random matrices typically have 
bounded RIC in some range of problem sizes (k,n,N). We provide the best known bound on the RIC for Gaussian 
matrices, which is also the smallest known bound on the RIC for any large rectangular matrix. Improvements over 
prior bounds are achieved by exploiting similarity of singular values for matrices which share a substantial number of 
columns. 
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1. Introduction. Interest in parsimonious solutions to underdetermined systems of equations 
has seen a spike with the introduction of compressed sensing [TTJ [7J [5] . Much of the analysis in 
this new topic has relied upon a new matrix quantity, the Restricted Isometry Constant (RIC), also 
referred to as the Restricted Isometry Property (RIP) constant. Let A be a matrix of size n x N 
and define the set of TV- vectors with at most k nonzero entries as 

(1.1) X N (k) ■■= {x € R N : ||2||«> < k}. 

Upper and lower RICs of A, U(k, n, N; A) and L(k, n, N; A) respectively, are defined as [51 [T] 

(1.2) U(k in ,N]A) := mine subject to (1 + c)\\x\\l > \\Ax\\\ Va; e x N {k)- 

c>0 



(1.3) L(k,n,N;A) := mine subject to (1 - c)||af||| < \\Ax\\l, Va; € X^[k); 

RICs differ from standard singular values squared in their combinatorial nature. U(k,n, N; A) 
and L(k, n, N; A) measure the maximum and the minimum deviation from unity (one) of the largest 
and smallest, respectively, square of the singular values of all ( . ) submatrices of A of size n x k 
constructed by taking k columns from A. The RICs can be equivalently defined as 

(1.4) U(k, n, N; A) := max X max (A* K A K ) - 1 

Kcn,\K\=k 

and 

(1.5) L(k, n, N; A) := 1 - min X min (A* K A K ) 

Kcn,\K\=k 

where ft := {1,2,..., N}, Ak is the restriction of the columns of A to a support set K C £1 with 
cardinality k (\K\ = k), and \ max (7£?) and X mm (B) are the smallest and largest eigenvalues of B 
respectively. 
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The standard notion of General Position is L(n, n, N; A) < 1, and Kruskal rank [19] is the largest 
k such that L(k, n, N; A) < 1. 

Many of the theorems in compressed sensing rely upon a "sensing matrix" having suitable bounds 
on its RIC. Unfortunately, computing the PJCs of a matrix A is in general NP-hard, [22]. Efforts 
are underway to design algorithms which compute accurate bounds on the RICs of a matrix, [TO] Q2] 
but to date these algorithms have a limited success, with the bounds only effective for k ~ n 1 / 2 . 
Lacking the ability to efficiently calculate the RICs of a given matrix, efforts are underway to 
compute probabilistic bounds for various random matrix ensembles. These efforts have followed 
three research programs: 

• Determination of the largest ensemble of matrices such that as the problem sizes (fc, n, N) 
grow, the RICs U(k, n, N; A) remains bounded and the L(k, n, N; A) bounded away from 1 

• Computing as accurate bounds as possible for particular ensembles, such as the Gaussian 
ensemble [HE], where the entries of A are drawn i.i.d. from the standard Gaussian Normal 
7V(0, 1/n). (In part as a model for i.i.d. mean zero ensembles.) 

• Computing as accurate bounds as possible for the partial Fourier ensemble [24] . where A is 
formed from random rows, j, or samples, ti, of a Fourier matrix with entries Fji = e 27rlJtl . 
(In part as a model for matrices possessing a fast matrix vector product.) 

This manuscript focuses on the second of these research programs, accurate bounds for the 
Gaussian/Wishart ensemble. Candes and Tao derived the first set of RIC bounds for the Gaussian 
ensemble using a union bound over all (^) submatrices and bounding the singular values of each 
submatrix using concentration of measure bounds [7J. Blanchard, Cartis and Tanner derived the 
second set of RIC bounds for the Gaussian ensemble, similarly using a union bound over all ( . ) 
submatrices, but achieved substantial improvements by using more accurate bounds on the proba- 
bility density function of Wishart matrices pQ. These bounds are presented here in Theorem |2 . 8| and 
Theorem |2. 10| respectively. This manuscript presents yet further improved bounds for the Gaussian 



ensemble, see Theorem 2.3 and Figure [271] by exploiting dependencies in the singular values of sub- 
matrices with overlapping support sets, say, Ak and Ak> with \K fl K'\ S> 1. These are the first 
RIC bounds that exploits this structure. In addition to asymptotic bounds for large problem sizes, 
we present bounds valid for finite values of (k,n, N). 

The manuscript is organised as follows: Our improved asymptotic bounds are stated in Section 
2 . 1 1 and their derivation described in Section |2.2[ Prior bounds are presented in Section |2.3| and are 
compared with those in Theorem |2.3| Bounds valid for finite values are presented in Section |2.4| 



A brief discussion on sparse approximation and compressed sensing and the implications of these 



bounds for compressed sensing is given in Section 2.5 Proof of technical lemmas used or assumed 
in our discussion come in the Appendix. 

2. RIC Bounds. We focus our attention on bounding the RIC for the Gaussian ensemble in 
the setting of proportional- growth asymptotics. 

Definition 2.1 (Proportional-Growth Asymptotics). A sequence of problem sizes (k,n,N) is 
said to follow proportional- growth asymptotics if 

(2.1) - = p n -> p and ^= = S n -> S for (5, p) e (0, l) 2 as (k, n, N) ->• oo. 

n N 

In this asymptotic we provide quantitative values, above which it is exponentially unlikely that 
the RIC will exceed. In Section |2.4| we show how our derivation of these bounds can also supply 
probabilities for specified bounds and finite values of (k,n,N). 

2.1. Improved RIP Bounds. The probability density functions (pdf) of the RIC for the 

Gaussian ensemble is currently unknown, but asymptotic probabilistic bounds have been proven. 
Our bounds, and earlier ones, for the RIC of the Gaussian ensemble built upon the bounds of the 
pdf's of the extreme eigenvalues of Gaussian (Wishart) matrices due to Edelman [TJ][T3]. All earlier 
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bounds on the RIC have been derived using union bounds that consider each of the (^f) submatrices 
of size n x k individually [2 |7j. We consider groups of submatrices where the columns of the 
submatrices in a group are from at most m > k distinct columns of A. We present our improved 



bounds in Theorem 2.3 preceded by the definition of the terms used in it given in Definition 2.2 
Plots of these bounds are displayed in Figure |2.1| 



Definition 2.2. Let (S 7 p) £ (0, l) 2 , 7 <E [p, S 1 ], and denote the Shannon Entropy with base e 
logarithms as H(p) :— pln(l/p) + (1 — p) ln(l/(l — p)). Let 



(2.2) 
(2.3) 

Define X r 

(2.4) 
(2.5) 



Vw (A, 7) := H (7) + - [ (1 - 7) In A + 7 In 7 + 1 - 7 - A 



(A, 7) := - ^ (I + 7) In A — 7017+1 + 7— A 



l (S, p; 7) and X max (S, p;j) as the solution to {2.4) and (2.5) respectively: 



<tyto» (X mm (5, p; 7), 7) + H(p6) - 5^ (p/ 7 ) = 0, for X min (S, p; 7) < 1 - 7, 
<5Vw (X max (S, p; j), 7 ) + H(pS)- 6jH (p/ 7 ) =0, for \ max (,5, p; 7) > 1 + 7. 



Let X mm (S,p) := max X mm (S, p; 7) and X max (S,p) := min X max {5, p; 7) and define 



(2.6) 



£ BT (6,p) :=1- X mm (S,p) and U" 1 (5, p) := X max {6, p) - 1 



BT 1 



That for each (5, p;j), (2.4) and (2.5) have a unique solution X mm (S, p;^) and X max {5, p; 7) 
respectively was proven in pQ. That X mm (S, p; 7) and X max (6, p; 7) have unique maxima and minima 
respectively over 76 [p, <5 _1 ] is established in Lemma 2.6 

Theorem 2.3. Let A be a matrix of size n x N whose entries are drawn i.i.d. from A/ ~(0, 1 /n). 
Let S and p be defined as in (2.1), and C BT (8,p) and U BT (5, p) be defined as in Definition 2.2 For 
any fixed e > 0, in the proportional- growth asymptotic, 



P(L{k,n,N) < C B1 (<5,p) +e) -> 1 and P(t/(fe, n, N) < U B1 (S, p) + e) ->■ 1 



exponentially in n. 

In the spirit of reproducible research, software and web forms that evaluate C BT (S,p) and 
U BT (S,p) are publicly available at [25] . 
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Sharpness of the bounds can be probed by comparison with empirically observed lower bounds 
on the RIC for finite dimensional draws from the Gaussian ensemble. There exist efficient algorithms 
for calculating lower bounds of RIC, [TH[TS]. These algorithms perform local searches for submatrices 
with extremal eigenvalues. The new bounds in Theorem |2.3[ see Figure 2.1 can be compared with 



empirical data displayed in Figure 2.2 




Fig. 2.2. Empirically observed lower bounds on RIC for A Gaussian. Observed lower bounds of L(k, n, N; A) 
(left panel) and U(k,n, N; A) (right panel). Although there is no computationally tractable method for calculating the 
RICs of a matrix, there are efficient algorithms which perform local searches for extremal eigenvalues of submatrices; 
allowing for observable lower bounds on the RICs. Algorithm for observing L(k,n, N), JT3j, and U(k,n, N), flffl , 
were applied to hundreds of A drawn i.i.d. Af(0, 1/n) with n = 400 and N increasing from 420 to 8000. 



To further demonstrate the sharpness of our bounds, we compute the maximum and minimum 
"sharpness ratios" of the bounds in Theorem |2.3| to empirically observed lower bounds; for each p, 
the maximum and minimum of the ratio is taken over all 5 G [0.05,0.9524], these are the same 5 
values used in Figure [272] These ratios are shown in the left panel of Figure [2~3| and are below 1.57 
of the empirically observed lower bounds on L(k, n, N) and U(k, n, N) observed with n = 400. 



- max(L(5,pVL(k,n,N)) 

- ma)!(U(S, P )/U(k 1 n,N)) 

- min(L(a, P )/L(k,n,N)) 
min|U|S,p)/U(k,n,N)) 



- ma),<L BCT /L BT ) 
ma>i(U BCT AJ BT ) 

- min(L BCT /L BT ) 

- min(U BCT /U BT ) 



XA (S o') C (S o') 

Fig. 2.3. Left panel: The maximum and minimum, over 5, sharpness ratios, ^ fc n jv a) anc ^ L(k n N A) as a 
function of p; with the maximum and minimum taken over all 8 £ [0.05,0.9524], the same 5 values used in Figure 
\2.2\ Right panel: The maximum and minimum, over S, improvement ratios over the previous best known bounds, 

U l4 BT (S p) an< ^ ^c BT (S p) as a f unc ti° n °f Pi with the maximum and minimum also taken over S £ [0.05, 0.9524]. 
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2.2. Discussion on the Construction of Improved RIC Bounds. The bounds in Theorem 
2.3 improve upon the earlier results of [T] by grouping matrices Ax and Ax> which share a significant 
number of columns from A. This is manifest in Definition |2.2| through the introduction of the free 
parameter 7 which is associated with the size of group considered. In this section we first discuss 
the way in which we construct these groups and the sense in which the bounds in Theorem 2.3 are 
optimal for this construction. Equipped with a suitable construction of groups, we discuss the way 
in which this grouping is employed to improve the RIC bounds from pQ. 

2.2.1. Construction of groups. We construct our groups of Ak by selecting a subset Mi 
from 1,2, ...,N of cardinality |M;| = to > k and setting Qi := UifcAf; \K\=k^-- The group Gi 
has (™) members, with any two members sharing at least 2k — m elements. Hence, the quantity 



7 = m/n in Definition 2.2 is associated with the cardinality of the groups Gi. In order to calculate 
bounds on the RIC of a matrix, we need a collection of groups whose union includes all (^) sets 
of cardinality k from f2 :— {1, 2, . . . , N}; that is, we need {Gi]^ = i such that G := U<=i &i with 
\G\ = (, ). From simple counting, the minimum number of groups Gi needed for this covering is at 

least r :— (^) (™) . Although the construction of a minimal covering is an open question |18] , even 
a simple random construction of the Gi's requires typically only a polynomial multiple of r groups, 
hence achieving the optimal large deviation rate. 

Lemma 2.4 ( |18j) . Set r = (, ) (™) and draw u := rN sets Mi each of cardinality to, drawn 
uniformly at random from the (^) possible sets of cardinality to. With G defined as above, 

(2.7) P (\G\ < (^\\ < C{k/N)N- 1 / 2 e- N{1 -^ 

where C{p) < f (2vrp(l - p))^ 1 / 2 ). 

Proof. Select one set K C fi of cardinality \K\ = k prior to the draw of the sets Mj. The 
probability that it is not contained in one set Mi is 1/r, and with each Mj drawn independently, 
the probability that it is not contained in any of the u sets Mi is (1 — r -1 )" < e~ u ' r . Applying a 
union bound over all (£) sets K yields 

Noting from Stirling's Inequality that 

(2.8) g(2*p(l -p)N)^e NH ^ < (T) < 4(2^(1 -p)Nr^e N ^\ 

with H(p) < In 2 for p £ [0, 1], and substituting in the selected value of u completes the proof. Note 
that an exponentially small probability can be obtained with u just larger than rNH(Sp), but the 
smaller polynomial factor is negligible for our purposes. □ 

Corollary 2.5. Given Lemma \2.4\ as n — > 00 in the proportional-growth asymptotics, the 
probability that all the (, ) k-subsets of 1, 2, . . . , N are covered by G converges to one exponentially 
in n. 

2.2.2. Decreasing the combinatorial term. We illustrate the way the groups Gi are used to 
improve the RIC bound on the upper RIC bound U(k, n, N; A); the bounds for L(k, n, N; A) following 
by a suitable replacement of maximizations/minimizations and sign changes. All previous bounds 
on the RIC for the Gaussian ensemble have overcome the combinatorial maximization/minimization 
by use a union bound over all (vj sets K C and then using a tail bound on the pdf of the extreme 
eigenvalues of A* K A K ; for some \* max > 0, 

P ( max \™*(A* K A K ) > X* max ) < ( N ) P (A mo * (A* K A K ) > X* max ) 

\KcSl,\K \=k J \k J 
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That the random variables X max (A* k Ak) are treated as independent is the principal deficiency of 
this bound. To exploit dependencies of this variable for K and K with significant overlap we exploit 
the groupings Gi, which, at least for m moderately larger than k, contain sets with significant overlap. 
For the moment we assume the groups {£/i}" =1 cover all K C O, and replace the above maximization 
over K with a double maximizations 

P ( max \ max {A* K A K ) > X* max ) = P ( max max X max {A* K A K ) > X* max ) . 

\Kcil,\K\=k J \i=l,...,u K<ZGi,\K\=k J 

The outer maximization can be bounded over all u sets Gi, again, using a simple union bound; 
however, with a smaller combinatorial term. The dependencies between X max (A* k Ak) for K C Gi 
can be incorporated in the bound by replacing the maximization over K C Gi by \ rnax (A* M . A^J 
where Mi is the subset of cardinality m containing all K C Gi, 

(2.9) P f . max max A™ Q *(A^) > X* max ) < uP (X max (A* M A M ) > X' max ) . 

\i=l,...,u KcQi,\K \=k J 



Selecting m = k recovers the usual union bound with u equal to ( k ) . Larger values of m decrease 
the combinatorial term at the cost of increasing X max {A* m Am) ■ The efficacy of this approach depends 
on the interplay between these two competing factors. In the proportional-growth asymptotic, this 
interplay is observed through the optimization over ^ = 7 e [p, S^ 1 ]. Definition |2.2| uses the 
tail bounds on the extreme eigenvalues of Wishart Matrices derived by Edelman |13] to bound 
p (\ max (A* M AM) > Knax)- The previously best known bound on the RIC for the Gaussian ensemble 
is recovered by selecting 7 = p in Definition 2.2 1]. The innovation of the bounds in Theorem 2.3 
follows from there always being a unique 7 > p such that X max (S, p; 7) is less than X max (5,p; p). 

Lemma 2.6. Given that X mm (S, p;j) and X max (8, p;j) are solutions to (2.4 ) and (2.5) respec- 
tively. For any fixed (6, p) there exist a unique 7 m i„ £ [p, <5 -1 ] which minimizes X max (S, p; 7) and 
a unique jmax € [Pi^ 1 ] which maximizes X mm (S, p; 7). Furthermore, ^min md j max are strictly 
larger than p. 



X-"(5,p;p). 
A.""(5,KY™) 



U BCT = X-"(5, K p)-l 



X-»(5,p;Y) 



X™(8,p;p) 



Y=P 




> Y 



Fig. 2.4. Left panel: Th e relat ionship between the new boundU BT (5, p) , Theorem \2.,3\ and the previous smallest 
id IA BCT {&, p), Theorem 2.10 where in the two bounds \ max (5, p;*y) is evalua ted at 7 = 7 m ; n and 7 = p 
respectively. Right panel: Th e relat ionship between the new bound C BT (6, p) , Theorem 2.3 and the previous smallest 

"/max and 7 = p 



bound C BCT (S,p), Theorem \2.10\ where in the two bounds \ mtn (S, p; 7) is evaluated at 7 
respectively. 



The optimal choices of 7 — p for U BT (p, 8) and C BT {p, 8) in (p, 8) £ (0, l) 2 is displayed in Figure 
2.5 The proof of Lemma 2.6 is presented in Appendix |3.1| 
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Fig. 2.5. Optimal choice of 7 — p for U BT (5, p) (left panel) andC BT (S,p) (right panel). 



2.3. Prior RIP Bounds. There have been two previous quantitative bounds for the RIC of 

the Gaussian ensemble in the proportional-growth asymptotics. The first bounds on the RIC of the 
Gaussian ensemble were supplied in [7] by Candes and Tao using union bounds and concentration 
of measure bounds on the extreme eigenvalues of Wishart Matrices from [20) . These bounds are 



stated in Theorem 2.8 with Definition |2.7| denning some of the terms used in the theorem and plots 



of these bounds are displayed in Figure |2.6| 

Definition 2.7. Let (S,p) € (0, l) 2 and define: 



U CT (5,p) := \l + ^p+(2S- 1 H(Sp)) 1 / 2 



and 



C CT (5,p) :=1 



l-^-(2S- 1 H(6p)) 



1/2 



Theorem 2.8 (Candes and Tao [?])• Let A be a matrix of size nx N whose entries are drawn 
i.i.d. from Af(0, 1/ti). Let 5 and p be defined as in (2.1), and £ CT (S,p) and U CT (S, p) be defined as 
in Definition \2. 

->• 1 



For any fixed e > 0, in the proportional- growth asymptotic, 
P(L(k,n,N) < £ CT (6 lP ) + e) -> 1 and P(U(k,n, N) < U CT '(5, p) 



exponentially in n. 

The bounds in Theorem 12.31 follow the construction of the second bounds on the RIC for the 
Gaussian ensemble, presented in pQ. Removing the optimization of 7 € [p, S^ 1 ] in Definition 2.2 
and fixing 7 = p recovers the bounds on L(k, n, N; A) and the first of two bounds on U(k, n, N; A) 
presented in [T]. The first bound on U(k, n, N; A) in PQ suffer from excessive overestimation when 
Sp ~ 1/2 due to the combinatorial term. In fact, this overestimation is so severe that for some (S, p) 
with Sp ~ 1/2, smaller bounds are obtained at (6, 1). This overestimation is somewhat ameliorated 



by noting the monotonicity of U(k,n, N; A) in k, obtaining the improved bound, see (2.12 1. These 
bounds are stated in Theorem |2.10| with Definition 2_^9 defining some of the terms used in the theorem 
and plots of these bounds are displayed in Figure 2.7 

Definition 2.9. Let (5, p) € (0, l) 2 , and denote the Shannon Entropy with base e logarithms as 
H(p) := pln(l/p) + (1 — p)ln(l/(l — p)). Let ip m i n (X, p) and ip max (A , p) be defin ed as in (2.2) and 
(|2.3| respectively. Define X min (S,p) and X max (S, p) as the solution to (2.10) and (2.11) respectively: 



(2.10) 



<^w(A™ n (5, P), P) + H(p5) = for X mm (5, p)<l-p, 
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(2.11) S^ max (X max (S,p) }P ) + H(p5) = for \ max (S, p) > 1 + p. 
Define C BCT {8, p) andU BCT {8,p) as 

(2.12) £ BCT (6,p) := 1- X min (S,p) and U BCT (S, p) := min X max (S, v) - 1. 




/// / 

4 <> 
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Fig. 2.7. U BCT {8,p) (left panel) and C BCT (S,p) (right panel) from Definition\E^for (<5, p) e (0, l) 2 . 

Theorem 2.10 (Blanchard, Cartis, and Tanner pQ). Let A be a matrix of size nx N whose 
entries are drawn i.i.d. from Af(0, 1/ n). Let S and p be defined as in (2.1|, and £ BCT (S, p) and 
U BCT (6, p) be defined as in Definition \2.(\ For any fixed e > 0, in the proportional- growth asymp- 
totic, 

P(L(k,n,N;A)<£ BCT (5,p)+e)^l and P(U(k, n, N; A) < U BCT (S, p) + e) ->■ 1 
exponentially in n. 

Figures |2.6| a nd |2.7| show that the bounds in Theorem |2.10| are a substantial improvement to 
those in Theorem |2.8[ The bounds presented here in Definition |2.2| and Theorem 2.3 are a further 



Bounds on Restricted Isometry Constants: Gaussian 



9 



improvement over those in [T], as implied by Lemma 2.6 

Corollary 2.11. Let C BT {5 1 p) and U BT {8, p) be defined as in Definition 
and U B (5, p) be defined as in Definition 2.9 For any fixed (S,p) £ (0, l) 2 , 



2.2 



id L 



BCT 



(M 



C BT (6,p) <£^ (M U^ (S,p)<U B ^ (S,p) 



-BCT 



BT I 



I BCT i 



The right panel of Figure |2.3| shows the ratio of the previously best known bounds, Definition 
2.9 to the new bounds, Definition 2.2 for each p, the ratio is maximized over S £ [0.05,0.9524]. 



2.4. Finite N Interpretations. The method of proof used to obtain the proportional-growth 
asymptotic bounds in Definition |2.2| also provides, albeit less elegant, bounds valid for finite values 
of (k, n, N) and specified probabilities of the bound being satisfied. For a specified problem instance 
(k, n, N) and e, bounds o n the p rob abilit ies P (U(k, n, N) > U BT {6, p) + e) and P (L(fc, n, N) > C BT (S, p) + e) 
are given in Propositions 2.12 and 2.13 respectively. 



Proposition 2.12. Let A be a matrix of size n x N whose entries are drawn i.i.d. from 
Af(0, l/?i). Define U BT (6,p) as in Definition 2.2 Then for any e > 0, 



>(u(k,n,N) > U(S n ,p n ) + ^ <p' max (n,\ max (8 n ,p n ))exp !ne (x™* (5 n , p n ) 



(2.13) 

where 



(2.14) 
and 

for ip ma x{Xl) define in p[ 



+ -(27rfe(l - fc/iV))- 1 / 2 exp(-iV(l - In 2)), 



1/2 2n~ 7 / 2 /5\ 3 /n7V( 7 -p)\ 1/2 



7A \4/ \l8(l-pS) 



H(p5)-5 1 H ( £ ) +<tyw(A, 7 ) 



Proposition 2.13. Let A be a matrix of size n x N whose entries are drawn i.i.d. from 
Af(0, 1/fi). Define C BT (5,p) as in Definition 2.2 Then for any e > 0, 



'(L{k,n,N) > L(5 n ,p n )+e) < p' min (n, X mm (5 n , p n )) exp Ine ■ (V" l (<5„, p n )\ 



(2.15) 

where 

(2.16) 

and 



+ -(27rfc(l - k/N))- 1/2 cxp(-iV(l - ln2)), 



V^(A, 7 ) :=^ 1 



5 \ 3 ey/X ( niV(7 — p) 



4/ ny/2 \jS(l- p6) 



1/2 



(?) 



H{p5) - 67H - +£VWn(A,7) 
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for -0mm (A, 7) define in 



The proofs of Propositions |2.12| and |2.13| are presented in Appendix |3.2| and also serve as the 
proof of Theorem |2.3| which follows by taking the appropriate limits. From Propositions |2.12| and 
2.13| we calculated bounds for a few example values of (k,n,N) and e. Table 2.1 shows bounds 
P (U(k, n, N) > U BT (S, p) + e) for a few values of (fc, n, N) with two different choices of e. It 



on f yu (K, n,i\ ) > U (o,p) + e 
is remarkable that these prob abilities are already close to zero for these small values of (k,n,N) 

shows bounds on P (L(k, n, N) > C BT (5, p) + e) for the same values 



and even for e < 1. Table 



2.2 



of (fc, n, N) as in Table |2.1[ but with even smaller values for e. Again, it is remarkable that these 
probabilities are extremely small, even for relatively small values of (fc, n, N) and e. 



fc 


n 


N 


e 


Prob 


100 


200 


2000 


10- 


-3 


2.9 x 10- 


-2 


200 


400 


4000 


10- 


-3 


9.5 x 10- 


-3 


400 


800 


8000 


10" 


-3 


2.9 x 10- 


3 


100 


200 


2000 


10" 


10 


3.2 x 10- 


2 


200 


400 


4000 


10" 


10 


1.1 x 10- 


-2 


400 


800 


8000 


10- 


10 


4.0 x 10- 


-3 



Table 2.1 

Prob is an upper bound ofP (U(k,n, N) > U BT (S,p) + e) for the specified (k,n,N) and e. 



fc 


n 


N 


e 


Prob 


100 


200 


2000 


10- 


5 


2.8 


x 10- 


-18 


200 


400 


4000 


10- 


5 


9.1 


x 10- 


-32 


400 


800 


8000 


10- 


5 


2.8 


x 10- 


-58 



Table 2.2 

Prob is an upper bound of P (i(fc, n, N) > C BT (8, p) + e) for the specified (k, n, N) and e. 



2.5. Implications for Sparse Approximation and Compressed Sensing. . 

The RIC were introduced by Candes and Tao [7] as a technique to prove that in certain con- 
ditions the sparsest solution of an undcrdctermined system of equations Ax = b (A of size n x N 
with n < N) can be found using linear programming. The RIC is now a widely used technique 
in the study of sparse approximation algorithms, allowing the analysis of sparse approximation al- 
gorithms without specifying the measurement matrix A. For instance, in [S] it was proven that if 
max(L(fc, n, N; A), U(k, n, N; A)) < y/2 — 1 then if Ax — b has a unique fc-sparse solution as its 
sparsest solution, then argmin ||z||i subject to Az — b will be this fc-sparse solution. A host of other 
RIC based conditions have been derived for this and other sparsifying algorithms. However, the 
values of (k,n,N) when these conditions on the RIC are satisfied can only be determined once the 
measurement matrix A has been specified [2]. 

The RIC bounds for the Gaussian ensemble discussed here allow one to state values of (fc, n, N) 
when sparse approximation recovery conditions are satisfied; and from these, guarantee the recovery 
of fc-sparse vectors from (A, b). Unfortunately, all existing sparse approximation bound on the RICs 
are sufficiently small that they are only satisfied for p <C 1, typically on the order of 10- 3 . Although 
the bounds presented here are a strict improvement over the previously best known bounds, and 
for some (S,p) achieve as much as a 20% decrease, see Figure 2.3 the improvements for p <C 1 are 
meager, approximately 0.5 — 1%. This limited improvement for compressed sensing algorithms is in 
large part due to the previous bounds being within 30% of empirically observed lower bounds on 
RIC for n = 400 when p < 1Q- 2 , [Q. 
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In [3], using RIC bounds from pQ, lower bounds on the phase transitions for exact recovery of 
k — sparse signals for three greedy algorithms and l\ — minimisation were presented. These curves 
are functions p S g(S) for Subspace Pursuit (SP) [S], Ps P (S) for Compressive Sampling Matching Pur- 
suit (CoSaMP) [23], pf\8) for Iterative Hard Thresholding (IHT) g] and p l HS) for h - minimisation 



[TB"] . Figure 1 in [3J shows a plot of these phase transition curves. Figure 2.8 shows the new phase 
transition curves based on our new bounds. The curves in the left Panel arc approximately 0.5 — 1% 
higher than those presented in pQ. 




0.1 02 03 04 05 0.6 0.7 0.8 0.9 




Fig. 2.8. Left panel: The lower bounds on the exact recovery phase transition for Gaussian random matrices 
for l\ — regularization, IHT, SP and CoSaMP implied by the RIC bounds in Theorem \2.3\ Right panel: The inverse 
of the phase transition lower bounds. 



3. Appendix. Here we present the proofs of the key theorems and lemmas stated in the paper. 
For other theorems and lemmas, especially technical lemmas used without stating in our analysis, 
you are referred to the Appendix of pQ. 



3.1. Proof of Lemma 2.6 , We start by showing that X max (5, p; 7) has a unique minimum for 
each fixed 6, p and 7 € [p, d^ 1 }. Equation (2.5) gives the implicit relation between \ max and 7 as 



where 



Hmaz {X max n) + H(pS) - &yH (p/i) = 0, for \ max > 1 + 7 , 



^ max (\ max , 7) = g [ (1 + 7) In (\ max ) - 7 In 7 + 1 + 7 - A f 



Therefore, — (X max ) = -, r In 

dj \ max - (1 + 7) 



7 



(7 - Pf 
3 



is equal to zero when 



(3.1) 



il-P? 



Let 7 m i„ satisfy (3.1). Since \ max > 1 + 7 > , J- (\ max ) is negative for 7 G [p,7mm), is zero at 



7n 



and is positive for 7 € (7min,<5 -1 ), equation (2.5) has a unique minima over 7 G [p, S^ 1 }, and 
the 7 that obtains the minima is strictly greater than p. 

Similarly, we show that A mm (<5, p; 7) has a unique maximum for each fixed 5, p and 7 €E [p, S -1 ]. 
Equation (2.4) gives the implicit relation between \ mm and 7 as 

5^ min (A mm , 7) + H(pS) - 5 1 H {p/i) = 0, for X mm < 1 - 7, 
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where 



Ipmin (A 



ff( 7 ) 



Therefore, — (A min ) = - 

dj K ' (1 -7) - \ r ' 



2 
In 



(1 - 7) In (A mm ) + 7 In 7 + I- 7- A" 1, 

is equal to zero when 



^3 , ymin 

(l- 7 ) 2 -(7-p) 2 



(3.2) 



7 3 -A mm = (l- 7 ) 2 (7-p) 5 



Let j max satisfy (3.2 1. Since < X mm < 1 + 7, j- (A mm ) is positive for 7 G [/9,7moa)) zer0 & t 7 



and negative for 7 e (7 ma n ( 5 1 ), equation (2.4) has a unique maxima over 76 [p, ^ 1 ], and the 7 
that obtains the maxima is strictly greater than p. 



3.2. Proof of main results, Theorem |2.3| and Propositions |2~. 12| and |2. 13[ . Here we give 
a proof similar to that given in pQ but we take great care at the non-exponential terms necessary 
for the calculations of bounds of probabilities for finite values of (k, n, N) in Section 2.4 We present 



the proof for U BT (5,p) in detail and sketch the proof of £ BT (S,p) which follows similarly. 

The following lemma of the bound on the probability distribution function of the maximum 
eigenvalue of a Wishart matrix due to Edelman, [TJ [31 [T3] is central to our proof. 

Lemma 3.1. (J 15$. presented in this form in f^j) Let Am be a matrix of size nxm whose entries 
are drawn i.i.d. from A/"(0, l/n). Let f m ax{m,n] A) denote the distribution function for the largest 
eigenvalue of the derived Wishart matrix A* m Am, of size m X m. Then f m ax(m, 1; A) satisfies: 



(3.3) fmax (m,n; A) < 



(27r)5(nA)~5 



n X 



i)r(f) 



e 2 



9max{m,n;X). 



It is helpful at this stage to rewrite Lemma |3.1| separating the exponential and polynomial parts 
(with respect to n) of g ma a;(w, u; A) as thus: 

Lemma 3.2. Let j n = m/n — >G [/Vm^ 1 ] and define 



■0mox(A,7) 



(1 + 7) In A - 7IX17+ 1 + 7 - A 



(3.4) 

Then 

(3.5) fmax(m,n;X) (m, n; A) < A; 7) ex P («■ • ip iiih.i (A, 7) 

where p m ax{n, A; 7) is a polynomial in n, X and 7, given by 



(3.6) 



Pmax(n, A; 7) 



1/2 



7 - 1 n- 7/2 A- 3 / 2 . 



Proo/. Let 7 „ = & and i h% ma:E (m, n; A)] = $i(m, n; A) + $ 2 (m, n; A) + $ 3 (m, n; A) where 
1 



$!(m,n;A) = — ln(2?r) 
2n 



~ ln(nA), $ 2 (m, n; A) = - [(1 + 7) ln(~) - A 



' m \ / n 
2/ V2 



1 / /? 

and $3(m, n; A) = In ^- 

We simplify <J> 3 (m, n; A) by using the second Binct's log gamma formulae 
(3.7) In (r(z)) > (z - 1/2) In z - z + In y/2n. 
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Thus we have 



$ 2 (m,n;X) + $ 3 (m,n;X) < - (1 + 7n ) In A - j n lwy n + 1 + j n - X -n 1 \n(nj n n 2 /2) 



Incorporating $i(m, n; A) and — n 1 In (7T7„n 2 /2) into p max {n, A; 7) and defining the exponent 



(3.4) as 



-0ma K (A,7):= lim — \n(g max (m,n; X)) = — (1 + 7) In A - 7 In 7 + 7 + 1 - A 

n— s-oo n I . 

completes the proof. □ 

The upper RIC bound U BT {8, p) is obtained by a) construct the groups Gi according to Lemma 
2.4 taking a union bound over all u = rN groups, and bounding the extreme eigenvalues within a 



group by the extreme eigenvalues of the Wishart matrices A* M _AMi„ see (2.9|. In preparation for 
bounding the right hand side of (2.9) we compute a bound on rNg max (m,n; A). 



From Lemma 3.2 and equation (2.8) we have 



(3.8) 



where 



and 



(3.9) 



2A V, T) (l) 9ma,[m,n; X) < p' max (n, A)e»^^ 



Vc/(A, 7 ) :=<5- 1 



H(p6)-5jH [ ) +<5V-w(A, 7 ) 



Pmax (n,X):=2X^-j _ p(5) ) Pmax(n,X; 7 ) 



The proof of proposition |2. 12 then follows. 
Proof. (Proof of Proposition 2.12) 



For e > with X max {8,p) = min X max (£, p; 7) being the optimal solution to (2.5|, 



1 (u(k, n, N) > U(5 n , Pn ) + e) = P (ll(k, n, N) > X max (<5„, p n ) - 1 + e) 

= p(l + U(k,n,N) > X max (6 n ,p n ) + e} 



(3.10) 



= N 



< N 



kj\k 

N\ ( m 
k 



A""»*(<5 n ,p n )+e 
— 1 />oo 



f m ax(m, n; X)dX 



g max (m,n;X)dX 



A m <"(5„,Pn)+e 



To bound the final integral in (3.10) we write g ma x(w,n;A) as a product of two separate functions 
- one of A and another of n and 7n , as g ma x{m, n; X) — <p(n, 7„)A~5 A? ( 1+ T™) e 2 where 

,„ ,1, .3 fn\ f (l+7n) 1 

V ,(n )7 „) = (2 7 r) I (n)-,y r(f 7n )r(f) - 

With this and using the fact that A ma:c (£ n , p n ) > l+7„ and that X%( 1+ln 'e~% > ' is strictly decreasing 
in A on [X max (5 n , p n ), 00) we can bound the integral in (3.10) as follows. 
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(3.11) 



g m ax(m,n;\)d\ < if(n,^ n )[X max (S n ,p n ) + e]^ 1 " 1 " 7 "^ - ^ 



X~?dX 



= [X max (5 n ,p n )}*g max [m,n;X max {5 n ,p n ) + e] 
= 2X max (S n ,p n )g max [m,n;X max (S n ,p n ) + e] 



A"*dA 



Thus (3.10) and (3.11) together gives 

p(u(k,n,N) > U(5 n , Pn ) + e) < 2X max (S n , p n )rNg max [m,n; X max (5 n , p n ) + e] 

< P' max (n, A mra ((5 n , P „)) exp [n • fa (X max (S n , Pn ) + e) 
(3.12) <p^(n,A ma3; ( ( 5„,p„))exp L- ^ 



where r = (^f) (™) , and the last inequality is due to fa(X) being strictly concave. □ 
The following is a corollary to Proposition [2T2] 

Corollary 3.3. Let (6,p) £ (0, l) 2 and let A be a matrix of size n x N whose entries are 
drawn i.i.d. from N (0,1/ 'n). Define U BT (5, p) = X max (5,p) — 1 where X max (8, p; 7) is the solution 
of \2.5\ for each 7 £ [p,(5 _1 ] and X max (S,p) := min X max (<5, p; 7) . Then for any e > 0, in the 



proportional- growth asymptotics 



P (U(k, n, N) > U BT {5, p) + e)^{) 



exponentially in n. 



Proof. From (3.12), since j^fa (X max (5 n , p n )) < is strictly bounded away from zero and all 
the limits of (6 n , p n ) are smoothly varying functions we conclude, for any e > 

lim P (U(k, n, N) > U BT {6, p) + e) -> 0. 

n—>oo 

□ 

Thus we finish the proof for U BT (S,p). We sketch the similar proof for Proposition 2.13 and 
C BT (8, p). Bounds on the probability distribution function of the minimum eigenvalue of a Wishart 
matrix are given in the following lemma. 

Lemma 3.4. (fT^jj, presented in this form in f^j) Let Am be a matrix of size nxm whose entries 
are drawn i.i.d. from Af(0,l/n). Let f m in{m,n;X) denote the distribution function for the smallest 
eigenvalue of the derived Wishart matrix A* m Am, of size m X m. Then f minim, ti; X) satisfies: 



2 J 



p(h±i : 



e " 2 =: g m in(m,n; X) 



Again an explicit expression of / m j„(m, n; A) in terms of exponential and polynomial parts leads to 
the following Lemma. 

Lemma 3.5. Let 7„ = m/n and define 



(3.14) 
Then 



Vw(A, 7 ) := if (7) + - [(1 - 7) In A + 7I117 + 1 - 7 - A 



(3.15) f m in{m, n; A) < g m in{m, n; A) X) exp [n ■ tp 

max (X,p,S;-y)) 
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where p m in(n, A; 7) is a polynomial in n, A and 7, given by 



(3.16) 



2tiV2A' 



The proof of Lemma |3.5| follows that of Lemma |3.2| and is omitted for brevity. Equipped with 
Lemma [3~5] a large deviation analysis yields 



(3.17) 



where 



and 



(3.18) 



N\ ( m 



2XN 



^l(A,7) := 8~ 



Am,n;\)<p' mm (n,\)e n ^ x ^ 



H( P 5)-8 1 H ( !- ) +<tyw(A, 7 ) 



f nN(j-p) \ 1/2 



Pmin 

(n,A). 



With Lemma 3.5 and (3.17), Propositio n |2. 13| follows similarly to the proof of Pr oposition 
stated earlier in this section. The bound L Bl (8, p) is a corollary of Proposition |2. 13 
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COROLLARY 3.6. Let (8,p) £ (0, l) 2 and let A be a matrix of size n x N whose entries are 
dra wn i .i.d. from J\f (0,1/ n). Define C BT (8,p) := 1 — \ m in(8,p) where \ m in(8, p;j) is the solution 

of (2.4) for each 7 <G [p, <$ _1 ] and X mm (S,p) := min X mm (6, p; 7). Then for any e > 0, in the 

1 

proportional- growth asymptotic 

P (L(k, n, N) > £ BT {6, p) + e)^0 



exponentially in n. 
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